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Seminar on National Intelligence Estimates 


Perennial questions about the validity and utility of 
National Intelligence Estimates (NIEs) again surfaced from 
the recent inquiry by the Center for the Study of Intelligence 
into the impact of intelligence on the foreign policy review 
and decision process. Policy makers have been critical of 
the relevance of NIEs and analysts have expressed concern 
that the goals of the estimative process are no longer 
clear. The establishment of the National Intelligence 
Council (NIC) as a collegial body to prepare and issue NIEs 
provided a timely occasion to take a critical look at estimative 
intelligence. 


On February 26, 1980 a group of 15 people met under the 
auspices of the Center for the Study of Intelligence to 
share their views and to exchange their ideas on the NIE 
process and its role for the future. Participants were 
drawn from Intelligence Community specialists who have been 
or who will be actively involved in estimative intelligence. 
The agenda was designed to provide for a discussion of the 
purpose, theory and practice of estimates. Particular 
emphasis was placed on the following questions: 


--What role should NIEs play in making foreign policy? 
--Is there an ideal methodology for writing an estimate? 


--Can an evaluation process to judge estimates be 
established? 


I, Historical Perspective 


At its inception, the National Intelligence Estimate 
was designed to serve as a comprehensive analytic product: 


--The NIE was to be a paper in which all of the 
available and relevant resources within the 
Community were brought to bear on the question(s) 
at hand. : 


--By projecting forward, the NIEs were to shed 
light not only on the situation as it existed, 
but also to make judgments about the likely course 
of events. This was not meant to be a prediction 
but rather a means of bringing out a range of 
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possible developments and the relative probabilities 
that they might occur. 


--NIEs were to provide a process for uncovering and 
illuminating real differences of opinion where 
they existed among the various agencies. They 
were'to provide, for example, an opportunity for 
the NFIB principals to dissent from the main 
paper. The fact that there were differences 
among the agencies or among analysts within a 
single agency was not important; rather it was’ 
an explanation of just what the arguments entailed 
which mattered most. 


Early Days 


In the early days of the Office of National Estimates, the 
original role and purpose of estimates was clear. Under the 
Eisenhower administration, estimates played an integral part 
in the policy formulation process. Each policy paper was 
required to have an estimate attached to it. The process 
also included an extraordinary effort to find out on a daily 
basis exactly what was bothering the President and what he 
really wanted to know. The NIEs were then carefully tailored 
to address those concerns. 


Viewing this 1950s period in hindsight, one participant 
felt that it would be more appropriate to label it as the 
"Golden Age of Estimates That Never Was.'' It was really 
during the Kennedy administration that estimates had an 
important policy impact, although “it was more unstructured 
and less deliberate. The DCI himsclf was actively 
involved in the estimative process during this period and 
had the confidence of the President. In addition, senior 
intelligence officers had access to the NSC principals, and 
i. able to provide useful and timely products directly to 
them. 


Losing the Path 
Whether due to organizational or personality problems, 
it-was the consensus of those present that we have somehow 
“Jost the path" toward effectual cstimates in recent years. 
Three reasons were suggested for this: 


--The vertical structure of analytic elements: 
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Policy makers want papers that provide a synthesis of 
materials reaching across diverse disciplines. They want 
intelligence analysis to put into perspective the various 
elements of a complicated situation. Such papers must 
provide the policy maker with an understanding of how these 
factors--political, economic, and technical--interact, and 
they must be written so that they can be rapidly read and 
absorbed. With the exception of the annual strategic and 
conventional forces estimates, there scems to be no real 
mechanism within the intelligence community today that can 
bring about this synthesis of facts and ideas. Instead 
the Intelligence Community provides '1000 bits of information 
stapled together--and thus 1000 pages to be read." 


--The United States as reactor: 


Within a rapidly changing international environment the 
United States has become less an initiator of action than a 
reactor to events. The structured style of policy making 
that predicates estimates on how others will react to U.S. 
action is thus obsolete. In such a "catch-up" situation 
there is some confusion about what the Intelligence Community 
should provide. Are policy makers so concentrated on the 
immediate, tactical problems that they miss or ignore the 
larger implications of their actions? This problem is 
reflected in the difficulty of getting anything written and 
read. beyond current intelligence. 


--The dispersion of skills formerly residing in ONE: 


Although there were many pluses and minuses to ONE, 
there seems little doubt that estimates it produced were used 
by policy makers during its existence. There is no single 
production office today that has the analytic skills the ONE 
Staff and Board were able to bring together during their 
heyday. 


One participant concluded this portion of the discussion 
by reminding his colleagues of the danger in generalizing 
too much or of putting the past into neat, well-defined 
little categories. Even in the past there was no instant or 
automatic line-up between an NIE and policy. Sometimes NIEs 
ran counter to policy; sometimes they had a potential for 
impact but were ignored by policy makers.. 
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II. NIE Role in the Policy Process Today 


_ What role should the NIEs be playing in the foreign 
policy process? Is it possible to construct them so that 
they fit the requirements of any given set of policy makers? 
For the present administration, tactical (short-range) 
decisions made within.a critical mode seem to have taken 
precedence over those with a more global (long-range) scope. 
If NIEs are intended to play a role in the more global 
aspects of decision-making, can they still be used effec- 
tively in crisis? : 


Several participants felt that NIEs have an important 
role to perform even in tactical situations. They cited the 
recent example of Panama as one in which an intelligence 
estimate provided important support in dealing with what was 
essentially a tactical policy problem. Yet many recognized 
that, as policy makers deal with short-range problems, 
traditional estimates are not always suitable. It appears 
that administrations become increasingly short-sighted as 
their time in office procecds, tending to concentrate 
more on the immediate problems that develop. 


Perhaps the tendency to shift from long-range to tactical 
policy making is one reason why the group could generally 
agree that Special National Intelligence Estimates have had 
the best track record over time. SNIEs were designed to 
address a narrow question over a finite period of time, to 
bring the issue into sharper focus and to use the best 
people available to accomplish this. The direct question/specific 
answer format did much to get the policy maker personally 
involved in the process. There was some concern among the 
participants that the SNIEs had become eroded over time and 
overridden by other systems. These “others"--the NIO system, 
for example--have not as yet been brought into the process 
sufficiently, and this allows the policy maker to go about 
his business without directly bringing in the Intelligence 
Community. One participant suggested that we should think 
about. returning to the SNIE in the future in. order to draw 
the Community back into the process and not just one individual 
or agency. 


It was thé consensus of those present, however, that to 
apply NIEs only to the actual decision stage of policy making 
is a much too narrow view of their usage. With the many 
potential intelligence inputs available, the NIE provides a 


SECRET 
Approved For Release 2004/10/28 :,CIA-RDP81B00493R000100020010-0 


Approved For Retease 2004/76/28 ! b1a-RDP81B00493R000100020010-0 


conceptual framework which has been argued out within the 
Community. It provides a carefully thought-through backdrop 
to help policy makers address problems as they occur. This 
instructive quality of estimates is important. 


The formal process of doing an estimate itself serves 
a twofold purpose: organization of substantive material and 
elimination or a "factoring out" of that which is unlikely 
to occur. If people. across the Community have done or are 
doing their jobs, the policy makers will be given: 


--a systematic view of the problem in all its aspects, 
-~-a checklist, and 


--an examination of relevant factors perhaps not 
thought about beforc. 


Whether the judgments to be drawn from this information 
are right or wrong is another question. Nevertheless, 
estimates ought to provide some assurance that the whole 
question has been thought about in depth. 


III. Usefulness of NIEs 


After a general acknowledgment that NIEs still do have 
a viable role to perform in the policy process, the seminar 
discussion turned its attention to their actual usefulness 
to the policy maker. Some consumers have complained that 
estimates are not relevant to a discussion of policy options 
for they do not provide clues to the options policy makers 
must eventually decide upon. Others complain that when 
NIEs do dircctly address policy options, it is in terms of 
"assailable generalities." 


In a potential crisis situation, such as El Salvador, 
to state in an estimate that the chances for the survival of 
the present junta are well less than 50% is an irrelevant 
judgment for the.policy maker who is already committed to a 
particular policy. He cannot readily abandon this policy 
oricntation without losing something of his credibility in 
the process. A judgment which says that his policy may be 
in serious trouble only, reminds the policy maker of the 
risks he has taken. 


SECRET 
5 


Approved For Release 2004/10/28 : CIA-RDP81B00493R000100020010-0 


; SECRET 
Approved For Release 2004/10/28 : CIA-RDP81B00493600100020010-0 


It is important to keep in mind that military estimates, 
notably NIE 11-3/8, are really quite different from other 
estimates where a political or economic context is the primary 
focus. The efficacy of the military estimates is less in 
dispute, but in fact their role is different from that of the 
“country/problem" estimates. In codifying a view of the state 
of adversary strategic or conventional forces, the predominant 
impact of these estimates is not at the higher policy making 
levels, although they may have an indirect effect there. 
Instead, they provide data to be used as basic reference 
material for all-level briefings and future planning.. One 
participant commented that one could corner the Secretary 
of State with a military estimate and get him to listen to 
it. Such technical, numbers-oriented estimates are more 
readily accepted because desk officers and policy makers do 
not know how to deal with such information on their own. 
Considering themsclves to be accomplished political analysts, 
policy makers feel capable of making just as good a judgment 
on the broader issues as an intelligence estimator. 


Intelligence by: Osmosis 


Information appears to move through the national 
security structure almost by osmosis. There is no stan- 
dardized way by which people are informed, particularly 
senior officials. Instead there are the informal luncheon 
conversations, the more formal staff-prepared papers and the 
oral bricfings. The result of all this is the welling up of 
vast amounts of intelligence as it moves through the structure 
in its various forms. Intelligence papers shape people's 
views often without their knowing it. Even if policy makers 
do not read a particular paper, they may still obtain its 
information from their colleagues or staffers with whom they 
talk. Such is the problem of assessing the usefulness of 
estimates--policy makers' criticism that NIEs "do not tell 
me anything that I already don't know" may in fact stem from 
this acquisition of NIE information via other sources. 


In remarking on the disrespect shown to NIIs in recent 
years one participant related the following story: A junior 
policy maker once told an intelligence officer wh ide noe. ger 
my estimatcs from you, I get them from The Washington POS tee 
Whereupon the intclligence analyst responded, “Ah, but where 
do you think The Washington Post got their estimates." 
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Based on his experiences, one discussant stated that 
many NIEs are being used at least in the sense that they are 
read, considered or modified with other inputs. Whether or 
not a policy maker agrees with them is another matter. He 
may consciously or unconsciously extract bits from them and 
put these into his own products. There are countless inputs 
into the policy process, and intelligence is only one of 
them. Within intelligence, NIEs are only a single component. 
Nevertheless, NIEs may have a secondary influence through 
their effect on the opinions of those who are to participate 
in the policy meetings. : 


Making Estimates Relevant: 


There are no casy solutions to making judgments about 
the impact of intelligence on policy makers and in determining 
whether an estimate is relevant to a policy maker's necds. 
Intelligence estimators can be more certain about the 
relevance of military estimates, but in the realm of politics, 
making such judgments is more difficult. Some discussants 
thought that we do not know as much as we should about what 
policy makers really want. 


One means of finding out is to increase our contact 
with consumers, particularly through the NIOs. Active 
dialogue must be established with the policy makers: 


--to detect the need for an estimate, 
--to make the estimate policy relevant, and 


--to enable the estimate to forecast to the point 
where it can give clues as to what policy makers 
should be considering in the future. 


The interaction between policy makers and intelligence 
is a critical part of the estimative process. But how are 
we to get policy makers more directly involved? Both INR 
and DIA make an effort to bring policy people into the 
estimates process by getting their inputs to the kinds of. 
questions estimates should be addressing, by discussing with 
them their concerns and by including them in the framing of 
the terms of reference. It is said that policy makers are 
always ''too busy." But is this the case?. INR, for example, 
“has routine access to policy makers through regular briefing 


hours. It is important to get as many as possible on the 
policy side involved in the terms-of-reference stage. If 
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policy makers believe that they have a stake in the estimate, 
it is more likely that they will be interested in the final 
product. 


The Quality of Estimates 


Although admittedly a generalization, it was felt that 
there had been a decline in the quality of the estimative 
product relative to the ability of the consumer to do his 
own estimating and intelligence work. In part this has been 
due to a growing sophistication of policy makers as well as 
their more direct access to raw intclligence traffic. If 
"knowledge is power," then perhaps the Intelligence Community 
has indeed lost something. 


There was some disagreement, however, concerning a 
second broad generalization, i.e. with the breakdown of ONE 
there has been a decline in the quality of people responsible 
for producing estimates. One participant felt that in 
actuality the reverse is more likely true. In its latter 
years, the Board and ONE included people who had nowhere 
else to go. Sent “out to pasture" in effect, they were not 
taken seriously by the senior management. Today our analysts 
are better trained, even though they have more limited 
experience at estimate writing. The important point is how 
the available people are used in constructing an estimate: 
one can follow the old "Greco-Roman" style with its rigid 
staff system or the freestyle, ad hoc arrangements found in 
more recent years. Despite complaints by consumers of 
irrelevance and:length, our estimative system is unique in 
its ability to handle competing sources of information. 


Iv. Improvements for NIEs 


It was suggested that perhaps the production of NIEs 
should follow more closely that of SNIEs, only placed in a 
broader context. As with SNIEs, policy people should 
be brought in to help form the terms of reference. As in 
the SNIIs, the questions to be addressed should be rigorously 
defined and focused. Such rigor is especially important 
where the consequences of U.S. policy decisions would be 
catastrophic if. wrong. Estimates should attempt to give 
alternative views and to include the implications of bad 
judgments. The NIE mechanism should provide to the policy 
maker not'’an answer but a range of possible answers. 


SECRET 


Approved For Release 2004/1 0/28": CIA-RDP81B00493R000100020010-0 


Approved For Release 200440285 CIA-RDP81B004938000100020010-0 


Is there some. methodological way to provide this range 
which is more “right" than others? Sherman Kent posited an 
analogy of the estimate process as a pyramid--the known and 
accepted facts making up its base with the point at the top 
being the agreed judgment about what is going to happen. In 
reality, however, this final, single point is rarely reached; 
perhaps we ought not to try. Perhaps it would be more 
useful to outline a scope of possibilities ranging from more 
than likely potential problems to those which can safely be 
ignored as unlikely. : 


NIE production currently tends to focus on assessments 
only of what the most likely course of action is or would be 
in the face of a crisis; policymakers are left unprepared 
for any others. There necds to be more attention to assessing 
the implications for the United States of alternative courses 
of action. The question arose as to whether or not such an 
assessment would in fact be an cstimate or some completely 
different paper. Most participants scemed to agree that we 
have the opportunity to do a lot more and still have it 
considered a legitimate cstimate. 


An excellent case study is a recent paper on the 
situation in El Salvador. As an estimate this paper goes 
beyond the basic judgment that the junta will fail. The 
paper states that barring any major outside intervention, it 
is inevitable that the revolutionary process will be triumphant 
and will profoundly alter the existing structures and relation- 
ships. It is the nature and duration of the revolutionary 
process itself which will determine the final outlook: 


--if the process takes a long time (e.g., two years) 
the final composition will be more extremely radical. 


--if the process is completed quickly (e.g. six months) 
the final picture will be more moderately left-wing. 


But nothing in the way of U.S. policy will make any 
difference. 


The question of what impact an estimate will have is an 
important consideration. NIlis can have the most impact where: 


--There is established an interactive network of policy, 
and intelligence contacts, 


--The DCI is actively involved, and 
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--the estimate does for the customer what he cannot 
do for himself. 


The policy maker's first reaction is "what can we do?" 
It is the job of NIEs to present the situation as it exists 
or will exist. It is however up to the policy maker to deal 
with it. NIEs can posit responses to U.S. policy options, 
but they cannot formulate those options. 


Caught up in daily operational routines, the policy 
maker may not be aware of some potential or developing crisis. 
For example, Iran did not receive early attention because 
policy makers and their staffs were concentrating on Arab 
and Isracli difficulties. NIEs must serve to keep the 
policy maker from getting side-swiped by unexpected events 
while attention is being diverted elsewhere. Here an added 
premium is placed on the personalities of the NIOs involved-- 
whether they are energetic jn dealing with policy makers, 
whether they have the respect of their policy counterparts 
and whether they are effective in presenting the estimates, 
especially ones which may be uncongenial to the policy 
maker's plans. 


Problems in Coordination 


There was some discussion about the [Ll Salvador example 
as an “estimate.'"' Although specific im Scope like an SNIE, 
it was produced as an NFAC product. This led directly into 
a comparison of the impact of NEAC versus interagency papers. 
The question here centered on the value accorded to our 
community coordination procedures. There js a common fecling 
among NIE consumers that coordination results in watered-down, 
rather useless consensus judgments. To be considered more 
useful, interagency estimates must not only lay out the existing 
disagreements but explain them as well. 


Part of the dissatisfaction with coordinated interagency | 
papers derives from their relative timeliness. Community. papers 
are slower, more protracted in their production. Agency papers, 
by comparison, are thought to be more up-to-date. This may, 
however, be attributable to reader prejudicc, as community 
Alert Memorandums are produced in a day. 


Regardless of whether or not an important paper is termed 
an “estimate,” it is up to the Community to sce that the 
right people are made aware of its existence. Unfortunately we 
tend to drop the ball at this point, relying almost on chance 
to sce that it is read. ; 
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V. Methodology 


Is the policy maker better served if we do alternative 
analysis or if we seek to reach agreed judgments? While 
acknowledging policy-maker criticism of coordinated judgments, 
a few of the seminar participants felt that there was a 
certain amount of myth to the merits of creative disagreement. 
Alternative’ judgments do have value but where a consensus 
can be reached it is good to say so. It need not, however, 
be an cither/or situation. It is possible to do alternative 
analysis and still reach some agreement. 


The 1973 Middle East war was cited as one example. It 
was the consensus of the intelligence community that the 
Arabs would not attack. Nonethiecss it was thought valid to 
consider the alternative hypothesis that they would attack. 
But the system did not operate to bring out this second 
"less probable" alternative. Perhaps this is in part due to 
the hammering out of the various positions by each group 
before the interagency session is convened. It is highly 
unlikely that real differences among analysts of one agency 
will surface in the process unless they also exist as 
differences of view among agencies. 


There was some disagreement as to the use of footnotes 
in estimates. One participant thought that the footnote 
procedure was pejorative, implying that there was some 
"rebel" out of phase with the group. Another participant 
felt that the footnote was useful in that differences of 
view not directly expressed in the text of the final estimates 
would stand out and be read. Appropriate footnoting would 
prevent a reader from otherwise missing these differences of 
opinion. 


It was generally agreed among the participants that 
alternative analysis should be used when possible and that 
it should be a mandatory cxercise to at least explore all 
alternatives. Where alternatives are rejected, they should 
still be listed with an explanation for their dismissal. 


VI. Evaluation of NIEs 


Without some means of evaluating the NIE process, it is 
difficult to tell how well we are doing. Looking at estimates 
after the fact can often present a skewed view. Asking the 

intended policy makers for their opinions on the quality of - 
the estimate does not provide a consistent standard, especially 
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when quality is equated with utility. One participant 
pointed out that it was important to make distinctions about 
the type of estimative judgments being cevaluated--lumping 
estimates which dealt with softer, more difficult information 
together with those using relatively unquestioned data 
distorts their respective contributions. 


One problem with postmortems conducted in the past was 
that the persons doing them were most likely to have been 
the analysts of the original estimates themselves. It is 
good to go back periodically to review an estimate or a 
serics of estimates to see exactly what they did or did not 
contribute, but it should be done by small teams, not the 
author. 


Postmortems of estimates whose original purpose was to 
undertake some kind of prediction do not help the policy 
maker. Such an evaluation will show only that the predicted 
event did or did not happen. Most policy makers already 
have some chosen objective in mind. What they most want to 
know from the estimate are the elements in the situation 
which would make the desired outcome more probable. It 
would be useful for policy makers if these types of estimates 
could be evaluated for future reference. 


Postmortems of 11-3/8 estimates are relatively casy to 
do-they are either right or wrong. The judgments contained 
in political estimates, however, are extremely difficult to 
evaluate. For one year a running box score was kept on the 
forecasting ability of these latter estimates. The results 
proved futile: 


--50% of the events were never resolved, and 


--in a substantial number of the remaining 50%, things 
predicted did indeed happen but not quite in the way 
described by the estimates. There were also a number 
of estimates which forecast such events as "the sun 
will rise tomorrow." 


This is not to say that postmortems of political estimates 
are useless. There are Jessons to be learned from hindsight. 
.One study analyzed existing estimates up-to. the 1960s. Pre-. 
dictions dealing with more quantitative analysis, such as 
technology or gross weapons capabilities,. proved to be 
adequate. But the thread of political reality in the more 
general estimates also proved to be surprisingly good: 75- 

80% right for the world as a wholc. 
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In the case of political estimates perhaps the question 
to be asked is not Did we predict accurately? but Did we 
understand the dynamics of the situation? It is, however, 
very difficult to sit down and read a paper five years after 
it was written and not compare it to the way we understand 
the situation now and judge its value accordingly. If the 
reader is able to get a sense of who is doing what to whom 
and the pressures involved, that is the best non-quantitative 
judgment that we can ask for. 


Much can be learned from an evaluation of a successful 
estimate--how it was drafted, by whom it was written and its 
terms of reference. Is there a certain constancy involved 
or is each one different? It may also help in hiring people 
to write future estimates if it is determined that a certain 
"type" of author is responsible for those judged to be most 
successful. 


VII. “Sumnary 


In the discussion of the role, theory and practice of 
NIEs, three major conclusions were reached by the seminar 
participants: 


1. It was:agreed that the NIE may play a greater role 
in the foreign policy process than policy makers realize or 
admit. The role of the NIE is not just-:as a vehicle for 
prediction but 


--it can educate the policy maker, 
--it can organize information for the policy maker, and 


--it can do for the policy maker what he does not want 
or cannot do for himself. 


2. It was agreed that not all conswner complaints 
about estimates are valid. There will always be some problem 
in the-way policy makers remember how they used the estimates 
as compared to how they in. fact may really have used them. In 
other words, “Intelligence officers remember their analysis as 
being more useful than it was while policy makers remember it 
as being less useful than it was." There are limited ways to 
overcome such complaints. One crucial factor is to -involve 
the policy maker in framing the terms of reference. This 
“helps to make the estimate more policy relevant while 
establishing for the policy maker some lcvel] of interest. 
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3. It was agreed that alternative methodologies are 
useful where they make sense. A consensus where it can be 
justly reached is good, but where these hypotheses are in 
competition alternative analysis is very useful. Footnoting is 


still a viable means of presenting alternative views or 
"thought. objections." 
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